Repository navigation
Conversation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Have the openshell client role publish the installed gateway's name, active TOML path, service scope, service user, and network, and add an openshell_test_gateway role that loads that context, resolves the CLI, restarts the gateway service, and collects its journal. A feature suite can then change the gateway's active configuration for its own run and restore it afterwards. Select the candidate runtime images for packaged installs through a qualification gateway.toml referenced by OPENSHELL_GATEWAY_CONFIG instead of OPENSHELL_* variables in the service environment file, so the gateway has a TOML file a suite can modify. Update the packaging asset check to match. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Define each CDI scenario once in vars/scenarios.yaml: the host fixtures it needs (directories, files, a character device, a socket), the CDI edits that reference them, and the behavior the workload is expected to observe. The cdi_fixtures role creates the fixtures, renders one tmachine.example/cdi-scenario spec with a device per scenario, and publishes the expectations as scenarios.json. Install the spec into /var/run/cdi from systemd-tmpfiles on every boot so it survives tmachine's setup, installation, and test disk layers, and run the role as part of the ubuntu-docker-rootful setup. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add a cdi feature suite that creates a sandbox for each CDI scenario by selecting its device explicitly with --gpu 1 and the Docker cdi_devices driver option, then checks what the workload observes or that creation is rejected. The sandbox only reports facts; the test makes every assertion, so a failure names the check that disagreed. Each scenario is its own test. The expectations record the behavior the sandbox has today, loaded from the scenarios.json published by the fixtures role. A OPENSHELL_CDI_CAPTURE mode prints observations instead of asserting, to record that behavior for a new scenario. The test playbook enables caller driver configuration in the installed gateway's active TOML for the run and restores it afterwards. Build the suite's nextest archive in tests/artifacts.nix, expose it as build-artifacts-cdi-tests, and run the suite on the Docker environment with the deb installer in the branch and release matrices. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Define the shared CDI context schema: the selected CDI device IDs and the workload-local paths at which a compute driver projects CDI spec directories. Carry it as an optional field in the sandbox boundary configuration, omitted from serialized output when absent so non-GPU bootstraps stay readable by older sandbox runtimes. No driver sets it yet. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Resolve the selected CDI devices against the spec directories projected into the workload and derive the filesystem requirements their edits imply: - device nodes, which need read-write access; - read-only mounts, expanded to the parent directory for shared libraries; - Unix sockets, kept as exact targets without filesystem grants; - the NVIDIA MPS pipe directory, accepted as a writable IPC directory only when mounted nosuid, nodev and noexec; - additional GIDs, excluding root. Effective mount access comes from the workload's mount table rather than spec options. Ordinary writable mounts, broad roots and conflicting uses of one path are rejected. Captured Orin, Spark and multi-GPU specs drive the production resolver against reviewed baselines. The resolver is wired into launch in the next commit. Signed-off-by: Evan Lezar <elezar@nvidia.com>
When the boundary configuration carries a CDI context, resolve it at agent start and use the result instead of the hard-coded GPU baseline: - add derived read-only paths, and read-write access for device nodes and IPC directories, unless the authored policy already covers them; - reject an authored read-only entry for a device node or IPC directory instead of overriding it; - validate every CDI target through the descriptor that anchors its Landlock rule at each launch and exec, rejecting broad-root aliases, type changes and changed mount access; - require Landlock enforcement for CDI targets even when the policy is best effort. Missing CDI additionalGids are reported as a warning, because some runtimes do not apply them and devices that do not depend on them stay usable. A required root group remains fatal. Enrichment, failures and the effective Landlock mode are reported as OCSF events. Sandboxes without a CDI context keep the existing GPU baseline. Signed-off-by: Evan Lezar <elezar@nvidia.com>
For GPU sandboxes, record the selected CDI device IDs and the spec directories the Docker daemon currently reports in the protected boundary configuration, and project those directories read-only under /.openshell/cdi-specs so the sandbox runtime can resolve the same specs. User mounts, image volumes and the workspace cannot overlap the projection path. Before creating the sandbox, probe each directory with a stopped container from the prepared workload image and skip directories that do not exist on the Docker host. Probes are bounded by a timeout, labelled for cleanup with their sandbox, and remove their anonymous volumes. A probe that cannot be removed no longer fails creation. With resource admission enabled, containers with a CDI context must keep a CDI device request matching the recorded selection and one read-only, non-recursive, private bind per recorded directory. Admission reads the bootstrap only for CDI containers, and boundary files are replaced atomically so it never observes a partial write. The Docker e2e gateway enables resource admission in GPU mode. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Describe how Docker GPU sandboxes derive filesystem access from the selected CDI devices, which CDI mounts are rejected, the NVIDIA MPS pipe directory exception, missing additionalGids handling on Docker releases before 29.5, and SELinux requirements for the projected spec directories. Add cluster debugging entries for CDI projection, admission, probe and restart failures, and document the CDI context in the core, sandbox and Docker driver READMEs. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Record the behavior of the named scenarios once filesystem access is derived from the selected CDI specs: read-only mounts and library parents are readable, device nodes are writable, and writable mounts, an unsafe library parent, and an NVIDIA MPS directory without noexec are rejected. Add scenarios for the MPS pipe directory with and without noexec. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Keep NVIDIA injection under Docker driver ownership and validate the persisted selection during admission and restart. Closes NVIDIA#4849 Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Thank you for your submission! We ask that you sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by adding a comment below using this text: I have read the DCO document and I hereby sign the DCO. You can retrigger this bot by commenting recheck in this Pull Request. Posted by the DCO Assistant Lite bot. |
I have read the DCO document and I hereby sign the DCO. |
WSL hardware validationTested implementation: A fresh qualification on Docker Desktop WSL2 passed for the explicit Environment: NVIDIA RTX Spark N1X, WSL Ubuntu 24.04 ARM64, kernel
Each CUDA probe returned: {"uid":998,"devices":1,"kernel_result":42,"cuda":"pass"}Enforcement probes confirmed denied writes outside permitted paths, denied direct network access, zero effective capabilities, The script initially attempted exec while gateway recovery still reported Provisioning and exited. After confirming that the same sandbox reached Ready, we continued the remaining probes successfully without recreating it or repeating the gateway restart. The early failed probe and successful continuation logs are retained separately. Raw qualification logs are retained on the test host under GitHub validation remains separate: Branch Checks and Helm Lint await maintainer |
|
recheck |
|
This pull request has had no activity for 14 days and is now marked stale. |
33350a8 to
baf423b
Compare
Summary
Support explicit all-GPU NVIDIA injection through ordinary Docker Desktop WSL sandbox creation. Preserve OpenShell ownership of the workload, protected GPU selection and restart admission while keeping existing CDI behavior unchanged.
Depends on #4204; the tested parent revision is
33350a881f55011286e7c3a68d8007e43e50e0bf.Related Issue
Closes #4849
Changes
docker.nvidia_wsl_gpus=all, limited to Docker Desktop WSL and rejecting conflicting CDI or counted GPU requests.Testing
Hardware evidence: #4837 (comment)
The test used the local uncommitted patch on the parent revision, identified by compressed patch SHA256
9791d6251efe4f958d460d62cb9cbc5f09646edcd0a1d0c5683455a580683c71. Local commitc0730ecb8c9c8a4562af534ca717af72a06ddd61matches the tested implementation patch; its only additional change is runtime-overview documentation. The SSH signature was verified locally. Hardware: RTX Spark N1X, Ubuntu24.04 WSL2, Docker Desktop4.96.0 / Engine29.9.1, NVIDIA620.66. Development gateway transport was used. Jetson, x86 hardware, individual GPU selection, model inference and production transport are not qualified by this evidence.Checklist
Draft review context
Opened as a draft following Jim Meyer's invitation to share the proposed approach for review, relayed by the author. Issue #4849 still awaits
state:accepted; the author currently has Read access to the upstream repository. The issue-acceptance check may remain red until maintainers record acceptance. This draft does not claim acceptance or merge readiness.The PR targets #4204's branch to isolate this change. Retarget and reconcile validation after the dependency lands; the closing reference takes effect when this work reaches the default branch.